{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5120ef0f-8510-4ad7-a0ea-2e306aec0064",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Transferring Files To and From a Remote Host\n",
    "\n",
    "Transfer a local file to a remote host's filesystem using Rsync via SSH.\n",
    "\n",
    "The following example transfers a file located on the remote host's filesystem at `/home/ubuntu/remote_unprocessed_file.png` to the local filesystem at `unprocessed_file.png` using Rsync via SSH."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1abfbc98",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "Upload a color image file to a remote host. Make note of the address of the remote host.\n",
    "\n",
    "### Procedure\n",
    "\n",
    "Make sure to swap the `private_key` and `remote_host` values with your own.\n",
    "\n",
    "1. Define an `Rsync` strategy with the remote host and private key path to be used for SSH."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "78667649",
   "metadata": {},
   "outputs": [],
   "source": [
    "import covalent as ct\n",
    "from pathlib import Path\n",
    "from typing import List, Tuple\n",
    "from skimage import io, color\n",
    "\n",
    "private_key = \"/path/to/private/key\"\n",
    "host_address = \"123.remote.host.address.com\"\n",
    "username = \"ubuntu\"\n",
    "\n",
    "unprocessed_filename = \"unprocessed_file.png\"\n",
    "processed_filename = \"processed_file.png\"\n",
    "\n",
    "unprocessed_filepath = str(Path(unprocessed_filename).resolve())\n",
    "processed_filepath = str(Path(processed_filename).resolve())\n",
    "\n",
    "remote_source_path = f\"/home/{username}/remote_{unprocessed_filename}\"\n",
    "remote_dest_path = f\"/home/{username}/remote_{processed_filename}\"\n",
    "\n",
    "rsync_strategy = ct.fs_strategies.Rsync(user=username, host=host_address, private_key_path=private_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb58d70",
   "metadata": {},
   "source": [
    "2. Generate the `FileTransfer` objects using `TransferFromRemote` and `TransferToRemote` factories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6753685b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_1 = ct.fs.TransferFromRemote(remote_source_path, unprocessed_filepath, strategy=rsync_strategy)\n",
    "ft_2 = ct.fs.TransferToRemote(remote_dest_path, processed_filepath, strategy=rsync_strategy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71910a29",
   "metadata": {},
   "source": [
    "The Covalent `Transfer*` functions intelligently assign the stage at which each file transfer should take place. The `TransferFromRemote` takes place before the electron is executed so that the electron can process the file. Conversely, the `TransferToRemote` takes place after the electron creates the outgoing file.\n",
    "\n",
    "Note that `TransferToRemote` is the only case in which the destination path is passed first, then the source. The `FileTransfer` object generated from it adheres to the `(<source_file_path>, <dest_file_path>)` convention. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1971b885",
   "metadata": {},
   "source": [
    "2. Define an electron, passing the Covalent `FileTransfer` objects to the `files` keyword argument in the decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4abc0a90",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ct.electron(files=[ft_1, ft_2]) # ft_1 is done before the electron is executed; ft_2 is done after.\n",
    "def to_grayscale(files: List[Tuple[str]] = None):\n",
    "\n",
    "    # Get the downloaded file's path\n",
    "    image_path = files[0][1] # destination filepath of first file transfer, downloaded before executing this electron\n",
    "    \n",
    "    # Convert the image to grayscale\n",
    "    img = io.imread(image_path)[:, :, :3] # limiting image to 3 channels\n",
    "    gray_img = color.rgb2gray(img)\n",
    "\n",
    "    # Save the grayscale image to the upload file path\n",
    "    gray_image_path = files[1][0] # source filepath of second file transfer, to be uploaded\n",
    "    io.imsave(gray_image_path, gray_img, mode=\"L\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2886440f",
   "metadata": {},
   "source": [
    "3. Create and dispatch a lattice to run the electron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "85e52654",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "COMPLETED\n"
     ]
    }
   ],
   "source": [
    "@ct.lattice\n",
    "def process_remote_data():\n",
    "    return to_grayscale()\n",
    "\n",
    "dispatch_id = ct.dispatch(process_remote_data)()\n",
    "status = ct.get_result(dispatch_id, wait=True).status\n",
    "print(status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57700440",
   "metadata": {},
   "source": [
    "Notes:\n",
    "\n",
    "- The transfer operations use `rsync` to perform the transfer. \n",
    "- In a typical real-world scenario, this kind of transfer can be used to move data generated by the workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244061a2",
   "metadata": {},
   "source": [
    "### See Also\n",
    "\n",
    "[Transferring Local Files During Workflows](./file_transfers_for_workflows_local.ipynb)\n",
    "\n",
    "[Transferring Files To and From an S3 Bucket](./file_transfers_to_from_s3.ipynb)\n",
    "\n",
    "[Transferring Files To and From Azure Blob Storage](./file_transfers_to_from_azure_blob.ipynb)\n",
    "\n",
    "[Transferring Files To and From Google Cloud Storage](./file_transfers_to_from_gcp_storage.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "vscode": {
   "interpreter": {
    "hash": "bc46502ba46b416dacb76460388e15e75b05e952c208e57c8294206d4336b33a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
